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Abstract: Although there are tools to assess student's readiness in an online learning context, little is known about the 
psychometric properties of the tools used or not. A systematic review of 5107 published and unpublished papers identified 
in a literature search on student online readiness assessment tools between 1990 and 2010 was conducted. The objective 
of this paper was to identify via a systematic review different tools allowing to assess the level of student's preparation in 
an online learning environment and which were published or not in scientific journals, and determine which of these tools 
have been validated. The results of the systematic review show that a standard tool does not exist, and that only ten 
instruments have been developed and published over the past 20 years to assess student's readiness. In addition, few tools 
published demonstrated good psychometric qualities, and many unpublished tools, considered as homemade tools, were 
internally developed in the universities by a team of professors without regard to their psychometric quality. Also, it 
appears that the tools that were published in scientific journals are rarely used by universities that offer online courses. 
Generally, the universities prefer to develop their own instrument that fits their online programs. 
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1 Introduction 

The recent studies have reported that the dropout rate of online students is higher compared to that of 
campus students (Dray & al., 2011). As the dropout rate remains high, it becomes a critical issue for online 
learning and a major concern for universities that offer online courses. Although there have been studies 
focused on development of the student online readiness assessment tools, they seem to have ignored an 
important detail about the psychometric quality of these instruments. Then, the objective of this paper is to 
identify via a systematic review the different tools that have been developed to assess online learning 
readiness and that have been psychometrically validated. 

2 Systematic review method 

In our study, we relied on the guide to conducting a systematic literature review prepared by Okoli & Schabram 
(2010), which is very comprehensive. 

2.1 Searching the literature 

The first step of a systematic review is to establish a research question that is clear and concise. In the context 
of this paper, the question chosen was: "What are the different student online readiness assessment tools?". 
To guide our research, we were assisted by a librarian specialist in identifying the major databases in the fields 
of education and of management and information systems. Nine electronic databases were searched: Aisel, 
CSA (ERIC, FRANCIS, Psychlnfo), EBSCO (Professional Development Collection, Business Source Complete), 
ED/ITlib, Education Abstracts (Wilson), Elsevier (ScienceDirect, Scopus), Emerald (Emerald Management Xtra), 
Sage, ProQuest (ABI / INFORMDateline, ABI / INFORM Global, ABI / INFORM Trade & Industry, CBCA 
Complete). In addition, an Internet search was conducted using Google Scholar. Also, other sources consisted 
of a hand-searching of relevant journals: Canadian Journal of Learning and Technology, Educational 
Technology and Society, European Journal of Open Distance and E-learning, International Journal of 
Instructional Technology & Distance Learning, International Review of Research in Open and Distance 
Learning, Journal of Educational Technology & Society, Journal of Interactive Media in Education, Journal of 
Interactive Online Learning, Journal of Learning Design, Language Learning and Technology, Turkish Online 
Journal of Distance Education, and International Journal of technologies in Higher Education. 

To identify all eligible studies on the development of measuring instruments in the context of e-learning, we 
used all terms obtained from the lexicon of e-learning. Identical terms in english are "cyber-training, distance 
education, distance learning, e-training, e-learning, e-university, Internet-based learning, Internet-based 
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training, Internet-delivered learning, Internet-delivered training, online education, online training, online 
course, online university, tele-education, teleteaching, virtual classroom, virtual learning, virtual university, 
Web-based education, Web-based Instruction, Web-based learning", and the French versions of these terms 
were also used: « apprentissage en ligne, apprentissage par Internet, apprentissage virtuel, etc. ». 

Others terms necessary for our research and their synonyms were also taken into account, namely, in English: 
"e-readiness, e-preparedness, e-preparation, predicting, success, instrument, development, scale, survey, 
questionnaire, tool", and in French: "developpement, instrument, echelle, outil, e-preparation, prediction, 
succes". We used two strings of information retrieval. The first string (string 1) is formed by three groups of 
keywords: 1) keywords referring to synonyms of e-learning "cyber-training, distance education, e-training", 2) 
keywords referring to the development of instruments "survey, scale", and finally 3) keywords referring to 
readiness "e-readiness, e-preparedness, e-preparation, etc.". The second string of information retrieval (string 
2) is the french version of string 1. 

Then, studies were included if they met the following four criteria: (1) studies published in journals, conference 
proceedings, and reports of expert groups, (2) studies aimed at the development of the measuring instrument 
in the context of online learning readiness, (3) editorials, books, theses, and studies done on a professional 
basis were not considered, (4) studies published in another language other than English or French were 
excluded, and (5) studies published from 1990 to 2010. 

2.2 Practical screen 

Thus, the electronic search identified 3544 citations as shown in Table 1, and the hand-search method resulted 
in 1563 additional articles. There were 258 duplicate references identified by t he EndNote X4 Bibliographic 
software that have been removed, and 5049 articles were excluded based on their titles and abstracts. After 
full text reading, 58 articles by two authors and 48 studies were excluded for not complying with the 
inclusion/exclusion criteria. Table 1 shows also that, after removing duplicate studies, the vast majority of 
studies selected are published in English (99.5%), as compared to those published in the French language, 
which represent only 0.5%. 


Distribution of 

the documents 

Electronic 

search 

Manual 

search 

Total 

Documents 

identified 

3802 

1563 

5365 

Duplicate 

documents 

258 

0 

258 

Single documents 

3544 

1563 

5107 

% in English 

99 

100 

99.5 

% in French 

1 

0 

0.5 


Table 1: Results of global search 

The exclusion of studies from the analysis was in most cases for the following reasons: 1) did not fit with the 
inclusion criteria, 2) not related to the question under review, 3) non-empirical studies (conceptual work, 
qualitative studies, etc.). Disagreements about study inclusion or exclusion were initially solved by consensus, 
and when this was not possible, they were arbitrarily resolved by a third reviewer. Only ten articles remained 
as shown in Figure 1. 
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Figure 1: Prescreening process of the articles 

2.3 Methodological quality and data extraction 

To assess the methodological quality of the ten studies retained, we were inspired by the evaluation grid 
established by Straub & al. (2004) concerning the evaluation of the quality of measuring instruments in the 
context of information systems. A quality evaluation of each of the ten articles was based on five criteria: 1) 
type of research, 2) content validity, 3) pre-test and/or pilot test, 4) construct validity, and 5) reliability. 
According to these authors, criteria 1 to 3 are highly desirable and recommended, but 4 and 5 are mandatory 
validities for instrument measurement. 

The quality of each study was carefully assessed using predefined criteria by Straub & al. (2004). The 
psychometric properties of the ten studies which were retrieved and which met the inclusion criteria are 
presented in table 2. 

Table 2: Methodological quality of the reviewed studies 


Authors 

fype of 
Research* 

Content 

validity 

Pretest 

/ 

Pilot 

test 

Construct 

validity 

Reliability 

Number of 

dimensions 
/Total scale 

Bernard & al. (2004) 

Exp 



V 

V 

4/9 items 

Kerr& al. (2006) 

Exp 



V 

V 

5/45 items 

Mattice & Dixon (1999) 

Exp 




V 

3/22 items 

Muse (2003) 

Exp 



V 

V 

7/26 items 

Osborn (2001) 

Exp 

V 

V 

V 

V 

6/20 items 

Parnell & Carraher (2003) 

Exp 

V 


V 

V 

3/9 items 

Pillay & al. (2007) 

Con 



V 

V 

4/18 items 

Roblyer & al. (2008) 

Exp 

V 


V 

V 

4/25 items 

Smith (2005) 

Exp 



V 

V 

2/12 items 

Watkins & al. (2004) 

Exp 

V 


V 

V 

6/27 items 


* Exp = Exploratory; Con = Confirmatory 
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2.3.1 Criterion 1: Type of the research 

As can be seen in Table 2, all these studies were based on exploratory research, except one (Pillay & al.) which 
used a confirmatory research. According to some authors, a confirmatory factor analysis is considered 
necessary to refine and validate the measurement scale, and is more efficient compared to the exploratory 
analysis in terms of validation of instruments (Straub & al, 2004). Confirmatory factor analysis (CFA) provides 
more information about the fit of the model, and has become accepted as the preferred way to assess 
construct validity (Bourque & al., 2006). 

2.3.2 Criterion 2: Pretest/Pilot test 

It appears that this criterion was rarely employed in theses studies, except in the study of Osborn that used a 
pilot study. Therefore, Boudreau & al. (2001) recommend that prior to the administration of the questionnaire 
to the sample selected for the study, it is necessary to pre-test the questionnaire. This criterion is largely 
neglected by researchers because of cost and time constraints. Then, the pre-test and / or pilot study is an 
important step in the process of construction and validation of an instrument (Straub & al., 2004). 

2.3.3 Criterion 3: Content validity 

The table 2 shows that the studies of Osborn, Parnell & Carraher, Roblyer & al., and Watkins & al. were based 
on literature review and a panel of two or three experts, without conducting an empirical test. The authors of 
these studies have chosen a face validity approach instead of a content validity to develop their tool, and 
Osborn has not detailed the evaluation procedure of the questionnaire items by the three experts and the 
method used. Furthermore, Penta & al. (2005) emphasize that face validity is the less stringent validation 
process, because it refers to the apparent value of the instrument rather than its real value. There are many 
methods to verify a content validity, like Delphi, Q-sort, and Content validity ratio. 

2.3.4 Criterion 4: Construct validity 

The above table shows that the principal component analysis (PCA) is widely used by the authors to determine 
the construct validity of their instrument, with the exception of Pillay & al. who opted for a confirmatory factor 
analysis (CFA). The vast majority of authors who have used the PCA method based themselves on Kaiser's 
criterion (eigenvalue equal or higher than 1) and CatteU's scree test to extract factors. However, Roblyer & al. 
did not specify in their study the strategies used to determine the number of factors to retain. Bernard & al. 
based themselves only on Kaiser's criterion, and Mattice & Dixon did not attempt to verify the construct 
validity of the instrument. For reasons of methodological rigor, many researchers recommend using a 
combination of criteria for determining the number of factors to retain, in addition to Kaiser's test, such as 
scree plot or Horn's parallel analysis (Yong-Mi, 2009). Among the studies retained, there is only one (Parnell & 
Carraher) that combined three methods (Kaiser's test, scree plot and parallel analysis) for determining the 
number of factors to retain. However, the maximum likelihood (ML) method is considered by many 
researchers as being an effective method to determine the number of factors to retain (Fabrigar & al., 1999). 

All the authors that have chosen the PCA method have used a varimax rotation. Although the orthogonal 
rotations produce solutions easier to interpret, many researchers agree on the fact that a rotation varimax can 
convey a distorted view of reality (Bourque & al, 2006). The latter added that solutions whose factors are 
completely independent, i.e. no correlation between factors, this is rarely the case in education. 

Also, some studies are based on factor loadings greater than or equal to 0.40, except the study of Kerr & al. 
which is based on factor loadings greater than or equal to 0.35, and Watkins & al. did not specify factors 
loading in their study. Hair & al. (2006) recommend that the factor loadings greater than 0.50 be considered 
practically significant. However, Roussel & al. (2002), strongly recommend to use a CFA to test the construct 
validity of an instrument. 

2.3.5. Criterion 5: Reliability 

As for this last criterion, it is found in Table 2 that the reliability data were limited to internal consistency and 
were consistently good to excellent across many studies. Nevertheless, internal consistency of some 
dimensions of the tools of Mattice & Dixon, Osborn, and Pillay & al. is below the threshold required by Hair & 
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al. (2006). For a confirmatory study, reliability should be equal to or above 0.70, and for exploratory study, 
equal to or above 0.60. For example, one of three factors of the tool of Mattice & Dixon had an internal 
consistency coefficient below 0.52; and for Osborn, three of six factors had a coefficient alpha below 0.50. 
However, t he Joreskog rho used in the confirmatory analysis is preferred to the Cronbach's alpha to test the 
reliability of internal consistency of the scale, since it is less sensitive to the number of items of a scale (Roussel 
& al., 2002). 

2.4 Synthesis of studies 

The present systematic review also showed that the tools vary in terms of the number of factors and items, the 
reliability of each factor, the process validation (type of research, content validity, pretest / pilot study, and 
construct validity), the size and the type of population, and the Likert type rating scales. However, some tools 
show little evidence of reliability and/or validity. Exploratory factor analysis, to a lesser extent, is also used to 
assess construct validity. However, the widespread use of principal component analysis for tasks theoretically 
conferred to the confirmatory factor analysis (validation instruments) must be questioned (Bourque & al., 
2006). Unfortunately, Fabirgar & al. (1999) emphasize that many researchers mistakenly believe that PCA is a 
type of EFA when in fact these procedures are different statistical methods designed to achieve different 
objectives. Also, some authors such as Bernard & al, Kerr & al. Muse, Osborn, and Roblyer & al. have carried 
predictive validity of their tool via different technical approaches like multi-regression analysis, discriminant 
analysis or binary logistic regression. This type of validity is considered to be optional and not mandatory to 
assess the psychometric quality of an instrument, but can also show the applied value of research (Straub & 
al., 2004). 

Thus, the total number of dimensions or factors used to measure readiness for online learning is 44. The 
longer tool containing more items is that of Kerr at al., with 45 items, and the shorter is that of Smith with 12 
items. Some dimensions identified from the ten tools appear quite similar (see Appendix A). For example, the 
items of the following dimensions: Computer skills, skills and relationships online, technology skills / mastery 
skills and the Web address all the issue of technical skills that a student should have before taking online 
courses. There are seven studies out of ten which have reported the percentage of variance explained by the 
factor solution. In contrast, among these studies, only three show a percentage higher than 60%, and in three 
others studies, a percentage less than 50%. In the social sciences, a factor solution that accounts for 60 percent 
or more of total variance is satisfactory (Hair & al., 2006). Another important point to consider is that the 
sample size in the majority of studies was very acceptable for factorial analysis except one. Eight studies out of 
ten had a subject to item ratio greater than 10:1; one had a ratio greater than 5:1, and one had a ratio less 
than 5:1. The minimal number of subjects providing usable data for the analysis should be ten participants per 
measured variable (Hair & al, 2006). The majority of studies were run in North America (7 in U.S. and 1 in 
Canada) and two others in Australia. The reported target population for all of the studies was students, except 
for the study of Watkins & al., where the target population was military people. 

The results of the systematic review show that the tools for assessing e-learning readiness are very important 
and essential for the student, and that their use may increase the retention rate in a virtual education, because 
they can identify students who are not prepared to follow online courses. However, it was found that the 
majority of these tools are rarely administered by universities to determine whether learners are able to 
succeed in a virtual learning environment. In contrast, some universities prefer to use their own instrument 
developed in-house (homemade) by a team of teachers. There are some tools for assessing e-learning 
readiness that are not identified by the systematic review because they are internally developed by a team of 
university professors without going through the validation process. 

A survey of eighteen universities offering online programs in the United States was conducted in April 2010 to 
check the psychometric quality of their self-assessment tools for online learners. The choice of these 
universities was conditioned by the fact that they had developed their own self-assessment instruments for 
students who wanted to take courses online. Some responses received from the responsible departments of 
distance education and / or the authors of the instruments show that these tools of self-assessment have not 
been validated or published. 

Although their psychometric properties are not shown, a majority of these tools should appear shorter and 
easier to use, and their content is more generic compared to those identified in the systematic review. There 
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are two aspects that differentiate unpublished tools from published ones in terms of type of scale and value of 
the scores. On the one hand, most of the unpublished tools were based on Likert’s scale of 3 points (e.g. 
Rarely/Sometimes/Always or Excellent/Good/Fair) or dichotomous scale (e.g. True/False or Yes/No). On the 
other, the scores obtained by these tools are arbitrary and subjective in the sense that they are based on 
judgment, because there is no compelling reason why they could not be set a little higher or a little lower. 

3 Discussion 

According to this review, we find that most of these instruments are old and less robust. The advantage of 
these free online self-assessment tools is to predict whether or no students are ready to take online classes, 
and to provide immediate feedback about the potential success of students in online learning environment. A 
high score obtained from these instruments is an indication that the student evidences some of characteristics 
of successful online learners. But the results from these self-assessment tools are subjective rather than 
objectives (measurable), and they may not provide the most accurate results unless more serious research is 
done that proves the validity and reliability of the instrument. 

We offer bellow some recommendations for advancing theory, measurement, and research in this area. This 
review shows that an e-learning readiness is a multidimensional construct that refers generally to computer 
Internet self-efficacy, self-direction, motivation, interaction, and attitude. Despite most authors of these tools 
agree on the multidimensionality of this construct, a systematic review reveals a lack of consensus about its 
dimensionality. The composition of this construct is questioned, and varies greatly from one study to the 
next.First, we strongly recommend clarify the concept of the e-learning readiness in order to promote greater 
terminological consistency. It is vital to be clear on the key constructs in the theoretical framework. Some 
authors have proposed a theory of e-learning readiness (Guglielmino and Gugliemino, 2003). This theory has 
two major components of learner readiness for successful e-learning: technical readiness and self-directed 
learning readiness. Each component is composed of specific knowledge, attitude, skills, and habits. In addition, 
we believe that some theories together (such as Moore's transactional distance theory, Bandura's social 
learning theory, Kember's open learning theory, and Information system success theory issued by DeLone and 
McLean) may help us to better conceptualize the construct of e-learning readiness in terms of its operational 
definition. 

In our second recommendation, we suggest that the researchers follow a rigorous methodology for developing 
a measurement instrument proposed by Bailey and Pearson (1983), Churchill (1979) and Moore and Benbassat 
(1991) and combined with recommendations of Boudreau et al. (2004) and Straub et al. (2004) with regard to 
the reliability and validity of the instruments. 

The last recommendation concerns the predictive validity of the instrument. We suggest use logistic regression 
or discriminant analyses to predict the probability of success and failure of the student in virtual courses. This 
type of validity is most important in terms of practical value to determine whether a student is ready or not for 
online learning. 

3.1 Limitations 

This review presents some limitations with regards to missing publications. The main limitation is the fact that 
only papers written in English or French were included in the selection process. Another limitation was the 
time period covered by the search (1990-2010). However, the data uncovered after this period, we cannot 
exclude the possibility that recent studies have used more rigorous methodology for developing an instrument 
valid and reliable to assess student readiness for online learning. Therefore, Baylor and Yorkston (2007) 
recommend that 'The reader can follow the search strategies outlined by the review authors to find related 
studies published more recently that might be of interest. For example, reviews will frequently list the keywords 
that were used in the searches’. 

4 Conclusion 

At the beginning of this work, little information was available about the tools for assessing e-learning readiness 
developed and used in universities. A systematic review was conducted and the content of each tool was 
analyzed. Existing tools were very diverse in terms of the type and number of dimensions. To our knowledge, 
no systematic review of tools for assessing e-learning readiness in the context of education has been 
undertaken and published to date. 


www.ejel.com 


380 


©Academic Publishing International Ltd 



Alem Farid 


Moreover, Bourque & al. (2006) have identified shortcomings, following an analysis of 1089 articles published 
during 1995-2005 in six Canadian periodicals in the field of education, in terms of presentation of factorial 
analysis and principal component analysis results with regard mainly to extraction and rotation methods. The 
same authors add that in most cases, researchers, including those in the field of education, use factorial 
analysis in an inappropriate way to assess the validity of psychometric instruments. 

Many unpublished or "homemade" tools have been developed, because some universities attempt to address 
the lack of a self-assessment tool which is standardized, reliable and valid to administer to students who want 
to enroll in online courses. However, scores obtained through these tools to assess the readiness of a student 
towards e-learning remain uncertain, because their psychometric properties have not been demonstrated, and 
these instruments were not developed according to a theoretical and empirical approach. 

This systematic review has identified the lack of standardization among these published and unpublished tools 
as a factor that could discourage the students from using them due to their heterogeneity. A valid and reliable 
student online readiness tool is very essential in order to identify students who are ready to take online 
courses, and to reduce withdrawal rate. 

This study is an effort to raise awareness among online educators and researchers of need to give attention to 
effective instrument design. 
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